Gallery of videos set to an audio time line

ABSTRACT

Machines, systems, and methods are provided for receiving, from a plurality of users, a plurality of messages, each message comprising audio content and video content taken by each of the plurality of users. For each message of the plurality of messages, the machines, systems, and method determine whether audio fingerprints of the audio content of the message correspond to a designated gallery comprising video content constructed to comprise a sequence of videos from a plurality of users set to an audio time line, and based on determining that the audio fingerprints of the audio content message correspond to the designated gallery, determine an audio segment of the audio time line of the designated gallery to which the audio content of the message corresponds. The machines, systems, and methods add the video content from the message to the audio time line of the designated gallery at the audio segment to which the audio content of the message corresponds.

REFERENCE TO RELATED APPLICATION

This application is a continuation of and claims priority of U.S. patentapplication Ser. No. 14/967,472, filed on Dec. 14, 2015, which is acontinuation of and claims priority of U.S. patent application Ser. No.14/578,271, filed on Dec. 19, 2014, which are hereby incorporated byreference in their entirety.

FIELD OF THE INVENTION

This invention relates generally to exchanging electronic messages in acomputer network. More particularly, this invention relates to a galleryof videos set to an audio time line.

BACKGROUND OF THE INVENTION

Mobile devices, such as smartphones, are used to generate messages. Themessages may be text messages, photographs (with or without augmentingtext) and videos. Users can share such messages with individuals intheir social network. However, there is no mechanism for automaticallyrouting a message with content related to a shared interest to a galleryof messages.

SUMMARY OF THE INVENTION

A machine includes a processor and a memory connected to the processor.The memory stores instructions executed by the processor to receive amessage with audio content and video content. Audio fingerprints withinthe audio content are evaluated. The audio fingerprints are matched toknown audio fingerprints to establish matched audio fingerprints. Adetermination is made whether the matched audio fingerprints correspondto a designated gallery constructed to receive a sequence of videos setto an audio time line. The matched audio fingerprints and correspondingvideo content are added to the audio time line. The operations arerepeated until the audio time line is populated with corresponding videocontent to form a completed gallery with video segments set to audiosegments that constitute a complete audio time line. The completedgallery is supplied in response to a request.

BRIEF DESCRIPTION OF THE FIGURES

The invention is more fully appreciated in connection with the followingdetailed description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 illustrates a system configured in accordance with an embodimentof the invention.

FIG. 2 illustrates a registration process utilized in accordance with anembodiment of the invention.

FIG. 3 illustrates a gallery collection with enrollment prompts utilizedin accordance with an embodiment of the invention.

FIG. 4 illustrates a message generation interface utilized in accordancewith an embodiment of the invention.

FIG. 5 illustrates server side processing performed in accordance withan embodiment of the invention.

FIG. 6 illustrates server side processing associated with anotherembodiment of the invention.

Like reference numerals refer to corresponding parts throughout theseveral views of the drawings.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 illustrates a system 100 configured in accordance with anembodiment of the invention. The system 100 includes a set of clientdevices 102_1 through 102_N and at least one server 104 connected via anetwork 106. The network 106 may be any combination of wired or wirelessnetworks.

Each client device 102 has standard components, such as a centralprocessing unit 110 and input/output devices 112 connected via a bus114. The input/output devices 112 may include a touch display, dedicatedcontrol buttons, physical connectors, speakers and the like. A networkinterface circuit 116 is also connected to the bus 114 and providesconnectivity to network 106. A memory 120 is also connected to the bus114. The memory 120 stores a communication application 122. Thecommunication application 122 includes instructions executed by CPU 110to coordinate communications with server 104 and/or other clientdevices. In particular, the communication application 122 is operativewith server 104 to exchange messages between client devices. The clientdevice may be in the form of a tablet, smartphone, wearable technology,laptop computer or desktop computer.

The server 104 also includes standard components, such as a centralprocessing unit 130, a bus 132, input/output devices 134 and a networkinterface circuit 136. A memory 140 is connected to the bus 132. Thememory 140 stores a communication module 142. The communication module142 includes instructions executed by the CPU 130 to coordinatecommunications with client devices 102. More particularly, thecommunication module 142 coordinates the exchange of messages betweenclient devices 102. The memory 140 also stores a gallery module 144. Thegallery module 144 includes instructions executed by the CPU 130 tostore messages from individuals with a shared interest or some otherassociation. Examples of types of associations contemplated byembodiments of the invention include photos or videos taken by usersthat include the same or similar object(s), such as guitar from aparticular manufacturer, a designer handbag, a soft drink or beer can,etc. The messages form a gallery, which may be supplied to a clientdevice 102 in response to a request from a client device 102. Thegallery may be a sequence of photographs and/or videos. Any of a textmessage, user drawings/annotations, animations, photo filters may beassociated with a photograph or video (e.g., as an overlay). In oneembodiment, the gallery is available for a specified transitory periodof time (e.g., a day, during an event, etc.).

FIG. 2 illustrates a process for registering users with common intereststhat desire to participate in posting to a gallery. This process may beimplemented by the gallery module 144. The gallery module 144 waits fora request for a gallery collection 200 from a client device 102. In theevent of a request (200—Yes), a gallery collection and enrollmentprompts 202 are sent from the server 104 to a client device 102.

Embodiments contemplate that a request could include an express requestsent by a user. For example, a user may request to participate in agallery by navigating to a ‘request’ screen from a user interface on thedevice display. In these examples, the user may select from any numberof existing galleries or create a new gallery by keying in or otherwisedescribing the gallery. Another example of an express request toparticipate in a gallery may include an action on the part of the userfrom an annotation feature offered from an image. For instance, after apicture or image has been taken, some embodiments allow a user to addtext, graphics, images, etc. to the picture/video. During thisannotation process, the input and detection of a particular key or swipemay be used to indicate an intent to add that picture/video to aparticular gallery. As an example, the use of a hashtag (“#”) or swipeadded to the text “Clippers” may be used to submit the picture/video toa Clippers gallery.

In addition to express user requests to submit content to a gallery,passive requests are also contemplated. For example, the detection of aparticular object in a picture or video (detected using, e.g., objectrecognition processes to detect an object or audio fingerprintingtechnologies to detect a song, etc.) could be used to prompt a user tojoin a gallery associated with the object or even automatically post toan existing gallery (e.g., with the user's consent). For example, if auser takes a picture or video of a Coca Cola bottle, upon detection ofthe bottle, the user may be asked if he or she wishes to participate ina Coca Cola gallery. Alternatively, the user may be asked if he or shewishes to post directly to a gallery without joining. In yet otherembodiments, the user's video or picture may be posted directly to thegallery (perhaps after receiving a blanket authorization at some pointearlier).

In additional embodiments, a revenue sharing process may be implementedto facilitate payments to the user for use of his or her image. Forexample, a user image may be added to a gallery sponsored by Coca Colaand in return for contributing to Coca Cola's gallery, a user mayreceive compensation on a flat-fee, per view or revenue share basis.

FIG. 3 illustrates an example of a gallery collection 300 displayed on aclient device 102. The gallery collection 300 may be supplied byactivating an icon (such as icon 402 of FIG. 4). Alternately, thegallery may be supplied in accordance with the examples provided above.The gallery collection 300 includes individually listed galleries, suchas 302 and an associated prompt 304. In this example, the prompt 304 isa radio button, selection of which results in a selected gallery.Hovering over or tapping the text description 302 of the gallery mayresult in the display of additional information about the gallery. Inthe case of “Clippers' Games” a gallery is formed during the course ofeach basketball game played by the Clippers. Thus, the gallery is arecurring gallery. Messages associated with the Clippers are postedduring the course of each basketball game. Thus, a collection ofmessages is available for the transitory duration of each basketballgame. Of course, other examples are also possible including availabilityfor a set amount of time after the event (e.g., twenty-four hours, oneweek, etc.).

The next example in the gallery collection 300 is a concert. This is asingle event gallery that lasts for a set amount of time after theduration of the event. The next example in the gallery collection 300 isfor a shared interest in an object, such as roses. This may be anongoing gallery that is used by registrants to post photographs orvideos of roses.

The next example in the gallery collection 300 is for the band Guns N′Roses. This gallery is constructed to receive messages from individualswith a shared interest in the band. In one embodiment, individual songsfrom the band may be selected. Thereafter, messages that reference orincorporate the individual songs may be posted to the gallery.

The collection may also include a theme such as midnight. Messagesgenerated on or around midnight are posted to the gallery. The messagesmay be identified by a timestamp associated with the message.

The collection may also include a theme such as “on the run”. Thisgallery is constructed to receive messages from client devices that arein motion (as measured by an accelerometer, gyroscope, GPS signal andthe like associated with the client device). Messages from such devicesare posted to the gallery.

The collection may also include a theme such as “I like it hot . . . ”to post messages when the temperature in a geolocation associated with amessage exceeds some threshold (e.g., 90° F.). The geolocation may bewhere the message is generated. The temperature may be obtained for thespecified geolocation by any number of online weather informationresources or from a thermometer device built into the device.

A final option shown in FIG. 3 is a request to create a new gallery.Activating the associated radio button 304 may result in a set ofprompts that allows a user to define a new gallery. The definition mayinclude a gallery name, keywords associated with the gallery, objectsassociated with the gallery, and the like.

Returning to FIG. 2, the gallery module 144 receives enrollment orparticipation requests 204. In response to such requests, a user isregistered with one or more selected galleries. Thus, a user registerswith one or more selected galleries 206. Thereafter, any messagerelevant to a selected gallery is automatically routed to the selectedgallery. This may be accomplished without the user separatelydesignating the gallery as a destination for the message. A singlemessage may be automatically routed to several galleries if the messagehas indicia relevant to each of the several galleries.

FIG. 4 illustrates a client device 102 that has taken a picture of arose 400. The communication application 122 and/or gallery module 144may be configured with instructions executed by the processor torecognize the rose (e.g., machine vision may be used to designate theobject as a rose). If the user is registered with the rose gallery, themessage is automatically routed to the rose gallery. An icon 402 may besupplied to annotate a message. Activation of the icon 402 may result ina text entry box 404 being supplied, along with a keyboard. The user maydesignate that the message includes a rose or may type in any other typeof message. A message recipient list may be invoked through the icon 402or through a swiping motion. The message recipient list is an existinglist of contacts for the user. Each contact selected will receive a copyof the message via routing through the communication module 142. Thus,for this message there are intentionally designated message recipients.However, depending upon the number of galleries that a user isregistered with, the message may be automatically forwarded to relevantgalleries without the user designating such galleries for the message.

FIG. 5 illustrates operations associated with an embodiment of thegallery module 144. The top of the figure illustrates a set of messagesenders, i.e., Sender_through Sender_N. Each message sender isassociated with a client device 102.

The server 104 receives messages 500 from any number of senders. Eachmessage is evaluated to determine whether message parameters correspondto selected galleries 502. For example, if the user has registered forthe roses gallery and generates the message shown in FIG. 4, then themessage corresponds to a selected gallery (502—Yes). Any number ofmessage parameters may be used. For example, in the case of the rosepicture in FIG. 4, the rose may be a designated object from machinevision operations performed by the gallery module. Alternately, or inaddition, text associated with the message may be a parameter that isevaluated for relevance to a selected gallery. For example, opticalcharacter recognition or word matching techniques may be used to link amessage with a gallery. Each gallery may have an associated collectionof key words and/or designated objects that are used to determinecorrespondence between a message and a gallery. Other message parametersmay include a time stamp (relevant to the midnight gallery), speedindicia (relevant to the “on the run” gallery) and temperature (relevantto the “I like it hot . . . ” gallery).

If the message does not have a parameter relevant to a selected gallery(502—No), then processing by the gallery module 144 is completed. Themessage is then routed in a standard manner by the communication module142. If the message does have a parameter corresponding to a selectedgallery (502—Yes), the message is posted to one or more galleries 504.

The gallery module 144 may include a curation interface that allows agallery administrator to optionally curate the gallery 506. For example,the curation interface may allow the administrator to deleteinappropriate or redundant messages. Alternately, machine vision andcontent rules may be used to automatically curate the gallery. Inembodiments that include compensation to be paid to a user, the curationinterface may be configured to make automatic payments to the users forthe use of their pictures or videos. The final operation of FIG. 5 is tosupply the gallery in response to requests 508 from any number of users.

Embodiments of the invention utilize galleries with specified transitoryperiods of time (e.g., a day, or during an event). Each gallery mayinclude individual messages shown in sequence, where the individualmessages are photographs or videos. Text may accompany a photograph orvideo, as shown in connection with FIG. 4.

An embodiment of the invention has a gallery with content set to anaudio time line. In some embodiments, the audio time line could be asong or a video clip. As discussed in connection with FIG. 3, a band mayhave a designated gallery with a selectable song. If a user registerswith a song then a message that incorporates the song may be posted tothe gallery. Embodiments of the invention utilize short (e.g., 3-6seconds) video messages. Therefore, a collection of messages can beconstructed for different segments of the audio time line associatedwith a song. As a result, a single song may have an associated mash-upof video segments. Observe here that the audio time line determines thevideo content. Typically, video content is used to establish a sequenceof videos. The use of the audio time line as a guiding factor results ininteresting and artistic combinations of videos.

Instead of a song, the audio time line may be a narrative, such as aspeech or a dialogue. For example, a “Gettysburg Address” gallery may beformed with a voice over of the famous speech by Abraham Lincoln. Videosegments that include words from the speech may be combined to populatea complete audio time line for the speech. Consequently, a voice over ofthe speech may have an associated mash-up of video segments.

There are known techniques for evaluating audio content. In particular,there are online services, such as Shazam® and Soundhound®, whichprovide real-time identification of songs and other audio tracks. Thegallery module 144 may be configured to access such online servicesand/or incorporate its own audio fingerprint identification techniques.The object recognition and fingerprinting operations disclosed hereinmay be performed on any of client device 102, server 104 or athird-party device (not shown).

As described above, embodiments contemplate that a user could becompensated by e.g., the audio time line owner or rights holder forhelping to create and promote a particular song or content. Thus, a userthat contributes to an audio time line could be compensated by someamount for each viewing of the time line.

FIG. 6 illustrates audio time line processing performed in accordancewith an embodiment of the invention. The gallery module 144 receivesmessages 600 from any number of message senders (Sender_1 throughSender_N). Each message is evaluated for audio fingerprints 602. Ifthere are no matches to known audio fingerprints (604—No), then thegallery module 144 processing is completed and the message is routed ina standard manner using communication module 142.

If matches with known audio fingerprints are established (604—Yes), adetermination is made whether there is a selected gallery with the sameaudio fingerprint. That is, has the message sender registered with agallery with the same audio fingerprint. If not (606—No), processing iscomplete. If so (606—Yes), the audio segment and corresponding videocontent are added to the audio time line 608. For example, the videomessage may have audio fingerprints for seconds 58 through 66 of a videotime line. The video message is then positioned at that location on thevideo time line.

It is then determined whether the audio Lime line is complete 610. Ifnot (610—No), the processing of the message is completed and the processmay wait for the submission of additional segments for possibleinclusion into the audio time line. If so (610—Yes), a gallery is formedwith video segments set to audio segments 612. The audio segments may bea master track of a song or narrative. Alternately, the audio segmentsfrom the videos themselves may be used. Different criteria may be usedto determine whether and audio time line is complete. All time slots inthe audio time line need not be filled.

The resultant gallery is then supplied to users in response to requests614. Thus, users may obtain crowd sourced video mash-ups set to an audiotime line.

Those skilled in the art will appreciate that the disclosed techniquesprovide improved technical results in the field of message combining andsharing. Messages are automatically shared with others that have acommon interest, as expressed through a registration process. After theregistration process, collections of messages are automaticallygenerated based upon automated evaluation of message content. Thisautomatic process may or may not be executed without a user interfaceand user input at the server. Thus, an autonomous message combining andsharing system is disclosed that omits the cost of a human operator andis able to operate as fast as a processor operates.

Embodiments of the invention include ephemeral galleries that reducepower consumption and administrative overhead. The ephemeral galleriesmay be configured for self-execution, for example, by setting messageparameters that will establish inclusion in a gallery and setting atransitory gallery period defined by an open time and a close time. Thegallery may be associated with a single ephemeral time period (e.g., inthe case of a concert) or a recurring ephemeral time period (e.g., inthe event of galleries for each basketball game played).

The crowd sourced galleries provide content from many perspectives. Anembodiment of the invention establishes innovative video mash-ups set toan audio Lime line as opposed to a flow of video images. Thus,establishing a novel technique for message combining.

An embodiment of the present invention relates to a computer storageproduct with a non-transitory computer readable storage medium havingcomputer code thereon for performing various computer-implementedoperations. The media and computer code may be those specially designedand constructed for the purposes of the present invention, or they maybe of the kind well known and available to those having skill in thecomputer software arts. Examples of computer-readable media include, butare not limited to: magnetic media, optical media, magneto-optical mediaand hardware devices that are specially configured to store and executeprogram code, such as application-specific integrated circuits(“ASICs”), programmable logic devices (“PLDs”) and ROM and RAM devices.Examples of computer code include machine code, such as produced by acompiler, and files containing higher-level code that are executed by acomputer using an interpreter. For example, an embodiment of theinvention may be implemented using JAVA®, C++, or other object-orientedprogramming language and development tools. Another embodiment of theinvention may be implemented in hardwired circuitry in place of, or incombination with, machine-executable software instructions.

The foregoing description, for purposes of explanation, used specificnomenclature to provide a thorough understanding of the invention.However, it will be apparent to one skilled in the art that specificdetails are not required in order to practice the invention. Thus, theforegoing descriptions of specific embodiments of the invention arepresented for purposes of illustration and description. They are notintended to be exhaustive or to limit the invention to the precise formsdisclosed; obviously, many modifications and variations are possible inview of the above teachings. The embodiments were chosen and describedin order to best explain the principles of the invention and itspractical applications, they thereby enable others skilled in the art tobest utilize the invention and various embodiments with variousmodifications as are suited to the particular use contemplated. It isintended that the following claims and their equivalents define thescope of the invention.

1. A machine, comprising: a processor; and a memory coupled to theprocessor, the memory storing instructions executed by the processor toperform operations comprising: receiving, from a plurality of users, aplurality of messages, each message comprising audio content and videocontent taken by each of the plurality of users; for each message of theplurality of messages: determining whether audio fingerprints of theaudio content of the message correspond to a designated gallerycomprising video content constructed to comprise a sequence of videosfrom a plurality of users set to an audio time line; based ondetermining that the audio fingerprints of the audio content messagecorrespond to the designated gallery, determining an audio segment ofthe audio time line of the designated gallery to which the audio contentof the message corresponds; and adding the video content from themessage to the audio time line of the designated gallery at the audiosegment to which the audio content of the message corresponds.
 2. Themachine of claim 1, the operations further comprising: sendingenrollment prompts to a user associated with a first message of theplurality of messages; receiving an enrollment request from the user forthe designated gallery; and registering the user with the designatedgallery.
 3. The machine of claim 2, wherein sending the enrollmentprompts to the user are in response to determining that the audiofingerprints correspond to the designated gallery.
 4. The machine ofclaim 2, wherein sending the enrollment prompts to the user are inresponse to an express request from the user to enroll in the designatedgallery.
 5. The machine of claim 1, wherein the designated gallery isavailable for a specified transitory period of Lime.
 6. The machine ofclaim 1, wherein the designated gallery is associated with a band, asong, a narrative, a speech, or a dialog.
 7. The machine of claim 1, theoperations further comprising: causing payment to be initiated to asender of a first message of the plurality of messages when the videocontent of the first message is added to the audio time line.
 8. Themachine of claim 1, wherein adding the video content from the message tothe audio time line of the designated gallery at the audio segment towhich the audio content of the message corresponds further comprisesadding the audio content of the message to the audio time line of thedesignated gallery at the audio segment to which the audio content ofthe message corresponds.
 9. The machine of claim 1, the operationsfurther comprising: providing the designated gallery to a computingdevice upon request from the computing device.
 10. Acomputer-implemented method comprising: receiving, from a plurality ofusers, a plurality of messages, each message comprising audio contentand video content taken by each of the plurality of users; for eachmessage of the plurality of messages: determining whether audiofingerprints of the audio content of the message correspond to adesignated gallery comprising video content constructed to comprise asequence of videos from a plurality of users set to an audio time line;based on determining that the audio fingerprints of the audio contentmessage correspond to the designated gallery, determining an audiosegment of the audio time line of the designated gallery to which theaudio content of the message corresponds; and adding the video contentfrom the message to the audio time line of the designated gallery at theaudio segment to which the audio content of the message corresponds 11.The computer-implemented method of claim 10, further comprising: sendingenrollment prompts to a user associated with a first message of theplurality of messages; receiving an enrollment request from the user forthe designated gallery; and registering the user with the designatedgallery.
 12. The computer-implemented method of claim 11, whereinsending the enrollment prompts to the user are in response todetermining that the audio fingerprints correspond to the designatedgallery.
 13. The computer-implemented method of claim 11, whereinsending the enrollment prompts to the user are in response to an expressrequest from the user to enroll in the designated gallery.
 14. Thecomputer-implemented method of claim 10, wherein the designated galleryis available for a specified transitory period of time.
 15. Thecomputer-implemented method of claim 10, wherein the designated galleryis associated with a band, a song, a narrative, a speech, or a dialog.16. The computer-implemented method of claim 10, further comprising:causing payment to be initiated to a sender of a first message of theplurality of messages when the video content of the first message isadded to the audio time line.
 17. The computer-implemented method ofclaim 10, wherein adding the video content from the message to the audiotime line of the designated gallery at the audio segment to which theaudio content of the message corresponds further comprises adding theaudio content of the message to the audio time line of the designatedgallery at the audio segment to which the audio content of the messagecorresponds.
 18. The computer-implemented method of claim 10, furthercomprising: providing the designated gallery to a computing device uponrequest from the computing device.
 19. A non-transitory computerreadable storage medium comprising instructions which, when executed byat least one processor, causes the at least one processor to performoperations comprising: receiving, from a plurality of users, a pluralityof messages, each message comprising audio content and video contenttaken by each of the plurality of users; for each message of theplurality of messages: determining whether audio fingerprints of theaudio content of the message correspond to a designated gallerycomprising video content constructed to comprise a sequence of videosfrom a plurality of users set to an audio time line; based ondetermining that the audio fingerprints of the audio content messagecorrespond to the designated gallery, determining an audio segment ofthe audio time line of the designated gallery to which the audio contentof the message corresponds; and adding the video content from themessage to the audio time line of the designated gallery at the audiosegment to which the audio content of the message corresponds
 20. Thenon-transitory computer readable storage medium of claim 19, whereinadding the video content from the message to the audio time line of thedesignated gallery at the audio segment to which the audio content ofthe message corresponds further comprises adding the audio content ofthe message to the audio time line of the designated gallery at theaudio segment to which the audio content of the message corresponds.